Corneal sphere tracking for generating an eye model

ABSTRACT

A head mounted display (HMD) comprises an eye tracking system configured to enable eye-tracking using light. The eye tracking system comprises two or more illumination sources positioned relative to one another and an optical detector in order to capture. The optical detector is configured to capture images of the cornea based on one or more reflections. The eye tracking unit is configured to generate a model of the user&#39;s eye. The generated eye model is used to determine eye tracking information such as gaze direction as the user glances at different objects in the HMD.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Application No. 62/306,758,filed Mar. 11, 2016 which is incorporated by reference in its entirety.

BACKGROUND

The present disclosure generally relates to eye tracking, andspecifically to corneal sphere tracking for generating an eye model.

Eye tracking is an important feature for head-mounted display (HMD)systems including systems used in virtual reality (VR) applications.Conventional tracking systems track features of the human eye and aretypically limited by the quality of the optical path. These conventionalsystems do not provide sufficient accuracy needed for eye tracking in aHMD system.

SUMMARY

An eye tracking system for generating an eye model is disclosed. The eyetracking system can be used in a VR system environment or other systemenvironments, such as an augmented reality (AR) system. The eye trackingsystem includes at least two illumination sources and an optical sensorfor modeling each eye. The optical sensor and the at least twoillumination sources are positioned relative to each other such that theoptical sensor is able to capture images of the illumination sourcesusing reflections from the cornea of the eye (hereinafter referred to as“corneal reflections”). The system gathers data as the user moves theireyes (e.g., during a calibration process that generates the eye modeland/or during normal use). The system models the eye by determiningradius and origin of a corneal sphere for each eye based on the gathereddata, where each eye is approximated as two spheres with one sphereapproximating a portion of a scleral surface of the eye and a portion ofthe other sphere (i.e., corneal sphere) approximating a portion of thecornea. The system uses the eye model to perform various optical actionssuch as determining the user's gaze direction, vergence angle/depth, andaccommodation depth.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system environment including a VR system,in accordance with an embodiment.

FIG. 2A is a diagram of a VR headset, in accordance with an embodiment.

FIG. 2B is a cross section of a front rigid body of the VR headset inFIG. 2A, in accordance with an embodiment.

FIG. 3 depicts an example eye tracking system for generating an eyemodel, in accordance with an embodiment.

FIG. 4 is a flowchart of an example process for generating an eye modelusing an eye tracking system, in accordance with an embodiment.

FIG. 5 is a flowchart of an example process for determining a user'sgaze direction using an eye model, in accordance with an embodiment.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles, or benefits touted, of the disclosure described herein.

DETAILED DESCRIPTION System Overview

FIG. 1 is a block diagram of a VR system environment 100 in which a VRconsole 110 operates. The system environment 100 shown by FIG. 1comprises a VR headset 105, an imaging device 135, and a VR inputinterface 140 that are each coupled to the VR console 110. While FIG. 1shows an example system 100 including one VR headset 105, one imagingdevice 135, and one VR input interface 140, in other embodiments anynumber of these components may be included in the system 100. Forexample, there may be multiple VR headsets 105 each having an associatedVR input interface 140 and being monitored by one or more imagingdevices 135, with each VR headset 105, VR input interface 140, andimaging devices 135 communicating with the VR console 110. Inalternative configurations, different and/or additional components maybe included in the system environment 100.

The VR headset 105 is a HMD that presents content to a user. Examples ofcontent presented by the VR head set include one or more images, video,audio, or some combination thereof. In some embodiments, audio ispresented via an external device (e.g., speakers and/or headphones) thatreceives audio information from the VR headset 105, the VR console 110,or both, and presents audio data based on the audio information. Anembodiment of the VR headset 105 is further described below inconjunction with FIGS. 2A and 2B. The VR headset 105 may comprise one ormore rigid bodies, which may be rigidly or non-rigidly coupled to eachother together. A rigid coupling between rigid bodies causes the coupledrigid bodies to act as a single rigid entity. In contrast, a non-rigidcoupling between rigid bodies allows the rigid bodies to move relativeto each other.

The VR headset 105 includes an electronic display 115, an optics block118, one or more locators 120, one or more position sensors 125, aninertial measurement unit (IMU) 130, and an eye tracking system 160. Theelectronic display 115 displays images to the user in accordance withdata received from the VR console 110. In various embodiments, theelectronic display 115 may comprise a single electronic display ormultiple electronic displays (e.g., a display for each eye of a user).Examples of the electronic display 115 include: a liquid crystal display(LCD), an organic light emitting diode (OLED) display, an active-matrixorganic light-emitting diode display (AMOLED), some other display, orsome combination thereof.

The optics block 118 magnifies received light from the electronicdisplay 115, corrects optical errors associated with the image light,and the corrected image light is presented to a user of the VR headset105. An optical element may be an aperture, a Fresnel lens, a convexlens, a concave lens, a filter, or any other suitable optical elementthat affects the image light emitted from the electronic display 115.Moreover, the optics block 118 may include combinations of differentoptical elements. In some embodiments, one or more of the opticalelements in the optics block 118 may have one or more coatings, such asanti-reflective coatings.

Magnification of the image light by the optics block 118 allows theelectronic display 115 to be physically smaller, weigh less, and consumeless power than larger displays. Additionally, magnification mayincrease a field of view of the displayed content. For example, thefield of view of the displayed content is such that the displayedcontent is presented using almost all (e.g., 110 degrees diagonal), andin some cases all, of the user's field of view. In some embodiments, theoptics block 118 is designed so its effective focal length is largerthan the spacing to the electronic display 115, which magnifies theimage light projected by the electronic display 115. Additionally, insome embodiments, the amount of magnification may be adjusted by addingor removing optical elements.

The optics block 118 may be designed to correct one or more types ofoptical errors in addition to fixed pattern noise (i.e., the screen dooreffect). Examples of optical errors include: two-dimensional opticalerrors, three-dimensional optical errors, or some combination thereof.Two-dimensional errors are optical aberrations that occur in twodimensions. Example types of two-dimensional errors include: barreldistortion, pincushion distortion, longitudinal chromatic aberration,transverse chromatic aberration, or any other type of two-dimensionaloptical error. Three-dimensional errors are optical errors that occur inthree dimensions. Example types of three-dimensional errors includespherical aberration, comatic aberration, field curvature, astigmatism,or any other type of three-dimensional optical error. In someembodiments, content provided to the electronic display 115 for displayis pre-distorted, and the optics block 118 corrects the distortion whenit receives image light from the electronic display 115 generated basedon the content.

The locators 120 are objects located in specific positions on the VRheadset 105 relative to one another and relative to a specific referencepoint on the VR headset 105. A locator 120 may be a light emitting diode(LED), a corner cube reflector, a reflective marker, a type of lightsource that contrasts with an environment in which the VR headset 105operates, or some combination thereof. In embodiments where the locators120 are active (i.e., an LED or other type of light emitting device),the locators 120 may emit light in the visible band (˜380 nm to 750 nm),in the infrared (IR) band (˜750 nm to 1 mm), in the ultraviolet band (10nm to 380 nm), some other portion of the electromagnetic spectrum, orsome combination thereof.

In some embodiments, the locators 120 are located beneath an outersurface of the VR headset 105, which is transparent to the wavelengthsof light emitted or reflected by the locators 120 or is thin enough notto substantially attenuate the wavelengths of light emitted or reflectedby the locators 120. Additionally, in some embodiments, the outersurface or other portions of the VR headset 105 are opaque in thevisible band of wavelengths of light. Thus, the locators 120 may emitlight in the IR band under an outer surface that is transparent in theIR band but opaque in the visible band.

The IMU 130 is an electronic device that generates fast calibration databased on measurement signals received from one or more of the positionsensors 125. A position sensor 125 generates one or more measurementsignals in response to motion of the VR headset 105. Examples ofposition sensors 125 include: one or more accelerometers, one or moregyroscopes, one or more magnetometers, another suitable type of sensorthat detects motion, a type of sensor used for error correction of theIMU 130, or some combination thereof. The position sensors 125 may belocated external to the IMU 130, internal to the IMU 130, or somecombination thereof.

Based on the one or more measurement signals from one or more positionsensors 125, the IMU 130 generates fast calibration data indicating anestimated position of the VR headset 105 relative to an initial positionof the VR headset 105. For example, the position sensors 125 includemultiple accelerometers to measure translational motion (forward/back,up/down, left/right) and multiple gyroscopes to measure rotationalmotion (e.g., pitch, yaw, roll). In some embodiments, the IMU 130rapidly samples the measurement signals and calculates the estimatedposition of the VR headset 105 from the sampled data. For example, theIMU 130 integrates the measurement signals received from theaccelerometers over time to estimate a velocity vector and integratesthe velocity vector over time to determine an estimated position of areference point on the VR headset 105. Alternatively, the IMU 130provides the sampled measurement signals to the VR console 110, whichdetermines the fast calibration data. The reference point is a pointthat may be used to describe the position of the VR headset 105. Whilethe reference point may generally be defined as a point in space;however, in practice the reference point is defined as a point withinthe VR headset 105 (e.g., a center of the IMU 130).

The IMU 130 receives one or more calibration parameters from the VRconsole 110. As further discussed below, the one or more calibrationparameters are used to maintain tracking of the VR headset 105. Based ona received calibration parameter, the IMU 130 may adjust one or more IMUparameters (e.g., sample rate). In some embodiments, certain calibrationparameters cause the IMU 130 to update an initial position of thereference point so it corresponds to a next calibrated position of thereference point. Updating the initial position of the reference point asthe next calibrated position of the reference point helps reduceaccumulated error associated with the determined estimated position. Theaccumulated error, also referred to as drift error, causes the estimatedposition of the reference point to “drift” away from the actual positionof the reference point over time.

The eye tracking system 160 generates the eye model using an examplecalibration process. The eye tracking system 160 includes an eyetracking unit and a control module. The eye tracking unit is locatedwithin the VR headset 105 and includes, among other components,illumination sources and optical sensors. The illumination sources(e.g., point light sources) and optical sensors (e.g., camera) of theeye tracking unit are used for corneal sphere tracking to determine amodel of an eye of a user while the user is wearing the VR headset 105.The illumination sources and the optical sensors are coupled to thecontrol module that performs the necessary data processing forgenerating the eye model and perform optical actions. The control moduleis located within the VR headset 105 and/or the VR console 110.

The imaging device 135 generates slow calibration data in accordancewith calibration parameters received from the VR console 110. Slowcalibration data includes one or more images showing observed positionsof the locators 120 that are detectable by the imaging device 135. Theimaging device 135 may include one or more cameras, one or more videocameras, any other device capable of capturing images including one ormore of the locators 120, or some combination thereof. Additionally, theimaging device 135 may include one or more filters (e.g., used toincrease signal to noise ratio). The imaging device 135 is configured todetect light emitted or reflected from locators 120 in a field of viewof the imaging device 135. In embodiments where the locators 120 includepassive elements (e.g., a retroreflector), the imaging device 135 mayinclude a light source that illuminates some or all of the locators 120,which retro-reflect the light towards the light source in the imagingdevice 135. Slow calibration data is communicated from the imagingdevice 135 to the VR console 110, and the imaging device 135 receivesone or more calibration parameters from the VR console 110 to adjust oneor more imaging parameters (e.g., focal length, focus, frame rate, ISO,sensor temperature, shutter speed, aperture, etc.).

The VR input interface 140 is a device that allows a user to send actionrequests to the VR console 110. An action request is a request toperform a particular action. For example, an action request may be tostart or end an application or to perform a particular action within theapplication. The VR input interface 140 may include one or more inputdevices. Example input devices include: a keyboard, a mouse, a gamecontroller, or any other suitable device for receiving action requestsand communicating the received action requests to the VR console 110. Anaction request received by the VR input interface 140 is communicated tothe VR console 110, which performs an action corresponding to the actionrequest. In some embodiments, the VR input interface 140 may providehaptic feedback to the user in accordance with instructions receivedfrom the VR console 110. For example, haptic feedback is provided whenan action request is received, or the VR console 110 communicatesinstructions to the VR input interface 140 causing the VR inputinterface 140 to generate haptic feedback when the VR console 110performs an action.

The VR console 110 provides content to the VR headset 105 forpresentation to the user in accordance with information received fromone or more of: the imaging device 135, the VR headset 105, and the VRinput interface 140. In the example shown in FIG. 1, the VR console 110includes an application store 145, a tracking module 150, and a VRengine 155. Some embodiments of the VR console 110 have differentmodules than those described in conjunction with FIG. 1. Similarly, thefunctions further described below may be distributed among components ofthe VR console 110 in a different manner than is described here.

The application store 145 stores one or more applications for executionby the VR console 110. An application is a group of instructions, thatwhen executed by a processor, generates content for presentation to theuser. Content generated by an application may be in response to inputsreceived from the user via movement of the VR headset 105 or the VRinterface device 140. Examples of applications include: gamingapplications, conferencing applications, video playback application, orother suitable applications.

The tracking module 150 calibrates the VR system 100 using one or morecalibration parameters and may adjust one or more calibration parametersto reduce error in determination of the position of the VR headset 105.For example, the tracking module 150 adjusts the focus of the imagingdevice 135 to obtain a more accurate position for observed locators onthe VR headset 105. Moreover, calibration performed by the trackingmodule 150 also accounts for information received from the IMU 130.Additionally, if tracking of the VR headset 105 is lost (e.g., theimaging device 135 loses line of sight of at least a threshold number ofthe locators 120), the tracking module 140 re-calibrates some or theentire system environment 100.

The tracking module 150 tracks movements of the VR headset 105 usingslow calibration information from the imaging device 135. The trackingmodule 150 determines positions of a reference point of the VR headset105 using observed locators from the slow calibration information and amodel of the VR headset 105. The tracking module 150 also determinespositions of a reference point of the VR headset 105 using positioninformation from the fast calibration information. Additionally, in someembodiments, the tracking module 150 may use portions of the fastcalibration information, the slow calibration information, or somecombination thereof, to predict a future location of the headset 105.The tracking module 150 provides the estimated or predicted futureposition of the VR headset 105 to the VR engine 155.

The VR engine 155 executes applications within the system environment100 and receives position information, acceleration information,velocity information, predicted future positions, or some combinationthereof of the VR headset 105 from the tracking module 150. Based on thereceived information, the VR engine 155 determines content to provide tothe VR headset 105 for presentation to the user. For example, if thereceived information indicates that the user has looked to the left, theVR engine 155 generates content for the VR headset 105 that mirrors theuser's movement in a virtual environment. Additionally, the VR engine155 performs an action within an application executing on the VR console110 in response to an action request received from the VR inputinterface 140 and provides feedback to the user that the action wasperformed. The provided feedback may be visual or audible feedback viathe VR headset 105 or haptic feedback via the VR input interface 140.

FIG. 2A is a diagram of a VR headset, in accordance with an embodiment.The VR headset 200 is an embodiment of the VR headset 105, and includesa front rigid body 205 and a band 210. The front rigid body 205 includesthe electronic display 115 (not shown in FIG. 2A), the IMU 130, the oneor more position sensors 125, and the locators 120. In the embodimentshown by FIG. 2A, the position sensors 125 are located within the IMU130, and neither the IMU 130 nor the position sensors 125 are visible tothe user.

The locators 120 are located in fixed positions on the front rigid body205 relative to one another and relative to a reference point 215. Inthe example of FIG. 2A, the reference point 215 is located at the centerof the IMU 130. Each of the locators 120 emit light that is detectableby the imaging device 135. Locators 120, or portions of locators 120,are located on a front side 220A, a top side 220B, a bottom side 220C, aright side 220D, and a left side 220E of the front rigid body 205 in theexample of FIG. 2A.

FIG. 2B is a cross section 225 of the front rigid body 205 of theembodiment of a VR headset 200 shown in FIG. 2A. As shown in FIG. 2B,the front rigid body 205 includes an optical block 230 that providesaltered image light to an exit pupil 250. The exit pupil 250 is thelocation of the front rigid body 205 where a user's eye 245 ispositioned. For purposes of illustration, FIG. 2B shows a cross section225 associated with a single eye 245, but another optical block,separate from the optical block 230, provides altered image light toanother eye of the user.

The optical block 230 includes an electronic display element 235 of theelectronic display 115, the optics block 118, and an eye tracking unit260. The electronic display element 235 emits image light toward theoptics block 118. The optics block 118 magnifies the image light, and insome embodiments, also corrects for one or more additional opticalerrors (e.g., distortion, astigmatism, etc.). The optics block 118directs the image light to the exit pupil 250 for presentation to theuser.

The VR headset 200 includes an eye tracking unit 260 (e.g., the eyetracking unit of the eye tracking system 160 of FIG. 1). The eyetracking unit 260 includes illumination sources and optical sensors. Inone embodiment, the eye tracking unit 260, as shown in FIG. 2B, includestwo illumination sources 262 and 264, and an optical sensor 266 for eacheye. The illumination sources and the optical sensor of the eye trackingunit 260 are coupled to a control module (not shown in FIG. 2B) thatperforms the necessary data processing for generating the eye model. Thecontrol module is located within the VR headset 105 and/or the VRconsole 110. Also, in some embodiments, there is at least one eyetracking unit 260 for the left eye of the user and at least one eyetracking unit 260 for the right eye of the user.

The illumination sources 262 and 264 and optical sensor 266 are used forcorneal sphere tracking of a user's eye. The eye tracking unit 260 ispositioned within the optical block 230 such that the optical sensor 266(e.g., camera) can capture images of the user's eye (and specificallycornea of the eye) over a range of eye motion. The illumination sources262 and 264 emit light such that when the emitted light reflects off ofthe user's eye while the user views the emitted light, the opticalsensor 266 captures one or more images of the user's eye. The eyetracking unit 260 is positioned within the optical block 230 such thatlight emitted from the illumination sources 262 and 264 reaches theuser's eye through the optics block 118. The eye tracking unit 260 maybe positioned either on-axis along the user's vision (e.g., as shown inFIG. 2B) or can be placed off-axis from the user's vision (e.g., to theleft of the optics block 118). An example corneal sphere tracking systemis described further below with reference to FIG. 3.

Eye Tracking System for Generating Eye Model

FIG. 3 depicts an example eye tracking system 300, in accordance with anembodiment. In some embodiments, the eye tracking system 300 is part ofthe eye tracking system 160 in the VR headset 105. In alternateembodiments, the eye tacking system 300 is part of some other device,e.g., a heads up display in an AR system, or some other system utilizingeye tracking. The eye tracking system 300 includes, among othercomponents, an eye tracking unit 360 and a control module 370. Forsimplification, the discussion of the eye tracking system 300 is withregard to a single eye of the user. However, in some embodiments,corresponding eye tracking units 360 may be employed for each of theuser's eyes. In such embodiments, a single control module 370 maycontrol the multiple eye tracking units 360.

The eye tracking unit 360 includes, among other components, two or moreillumination sources (e.g., illumination sources 362 and 364) and one ormore optical sensors (e.g., optical sensor 366). The illuminationsources (e.g., point light sources) and optical sensors (e.g., camera)of the eye tracking unit are used for corneal sphere tracking and todetermine a model of an eye of a user while the user is wearing the VRheadset 105. The illumination sources 362 and 364 have well-knownemission characteristics such as ideal point light sources. In oneembodiment, two illumination sources are used. Alternatively, more thantwo illumination sources, such as, a ring of illumination sources areused. For example, the ring of illumination sources can be positionedeither in the same two-dimensional plane or in arbitrary positionsrelative to a reference point (e.g., location of an entrance pupil ofthe HMD or reference point 215). In one embodiment, the illuminationsources can be located outside of the user's line of sight. Illuminationsources positioned arbitrarily from the reference point can be placed atdifferent depths from the reference point and/or at non-uniform spacingbetween the sources to improve the accuracy of the eye tracking.

In some embodiments, the two or more illumination sources comprisedifferent characteristics for either all of the illumination sources orbetween the illumination sources. For example, light originating fromthe two or more illumination sources can include one or more of:different wavelengths, modulated at different frequencies or amplitudes(i.e., varying intensity), have different temporal coherence thatdescribes the correlation between the two light waves at differentpoints in time, and multiplexed in either time or frequency domain.

The optical sensor 366 captures images of the user's eye to capturecorneal reflections. For example, the optical sensor 366 is a camerathat can capture still pictures or video. The optical sensor 366 has aplurality of parameters such as focal length, focus, frame rate, ISO,sensor temperature, shutter speed, aperture, resolution, etc. In someembodiments, the optical sensor 366 has a high frame rate and highresolution. The optical sensor 366 can capture either two-dimensionalimages or three-dimensional images. The optical sensor 366 is placedsuch that the corneal reflections in response to the light from theillumination sources incident upon the eye can be captured over a rangeof eye movements (e.g., a maximum possible range). For example, when aring of illumination sources are placed around the eye, the opticalsensor 366 is placed pointed towards the eye around the center of thering (e.g., in the line of sight of the user). Alternatively, theoptical sensor 366 is placed off-axis such that it is outside of themain line of sight of the user. In one embodiment, more than one opticalsensor 366 can be used per eye to capture corneal reflections of the eyewhile light from illumination sources is incident upon the eye. Theoptical sensor 366 may be a detector that can measure a direction ofcorneal reflections such as column sensors, waveguides, and the like.

The illumination sources 362 and 364 emit light that is reflected at thecornea, which is then captured (e.g., as an image) at the optical sensor366. For example, light rays represented by arrows 362-I and 364-Ioriginating at the illumination sources 362 and 364 are incident uponthe eye. When light is incident upon the human eye, the eye producesmultiple reflections such as a reflection from the outer surface of thecornea and another reflection from the inner surface of the cornea. Inone embodiment, the optical sensor 366 captures the reflected light fromthe outer surface of the cornea in the captured images. For example,reflected light represented by arrows 362-R and 364-R is captured inimages captured by the optical sensor 366. Alternatively oradditionally, the optical sensor 366 captures the reflected light fromthe inner surface of the cornea. The reflections from the cornea (e.g.,from inner surface and/or outer surface) are herein referred to ascorneal reflections. An example process of generating an eye model usingcorneal sphere eye tracking is further described below with reference toFIG. 4.

The control module 370 generates eye models and performs opticalactions. For example, the control module 370 performs the calibration togenerate an eye model for one or both of a user's eyes. In someembodiments, a single control module 370 may control the multiple eyetracking units 360 such as one eye tracking unit 360 for the left eyeand another eye tracking unit 360 for the right eye.

An example calibration process, described below in conjunction with FIG.4, includes the steps of turning on the illumination sources 362 and364, capturing images including corneal reflections at the opticalsensor 366 while the user is viewing known locations on the VR headset105, and further processing of the captured images to generate an eyemodel. An example normal mode of operation, described below inconjunction with FIG. 5, includes capturing corneal reflections at theoptical sensor 366 while the user is viewing content on the VR headset105 in the normal mode of operation and processing of the capturedimages to perform one or more optical actions such as determining auser's gaze direction. The control module 370 is located at the VRheadset 105 and/or the VR console 110. The control module 370 is coupledwith the eye tracking unit 360 such that the illumination sources 362and 364, and optical sensor 366 can communicate with the control module370.

The eye tracking system 300 generates a model for the user's eye. In oneembodiment, the user's eye is modeled as two spheres 305 and 310 ofdifferent radii, where sphere 305 approximates a portion of the scleralsurface of the eye (e.g., overall eye) and a portion of sphere 310approximates the cornea of the eye. The center (or origin) of the sphere305 is represented by point 306 and the center of corneal sphere 310 isrepresented by point 311. Element 315 represents the lens of the eye. Inother embodiments, the cornea may be modeled as a complex surface.

The human eye can be modeled using two spheres with a bigger sphere(i.e., sphere 305) representing an approximation a portion of thescleral surface of the eye (e.g., overall eye) and a smaller sphere(i.e., sphere 310) representing an approximation of a portion of thecornea of the eye, where the two spheres have different radii and theircenters are offset from each other. While it is known that the corneaforms only a small curved portion in the eye and is not a sphere in andoff itself, the cornea can be approximated as a portion of a sphere. Inone embodiment, the sphere 305 has a radius of approximately 25 mm andcorneal sphere 310 has a radius of approximately 8 mm. The centers ofthe sphere 305 and sphere 310 are offset from each other as shown inFIG. 3. When the eye rotates while viewing content on a head mounteddisplay (e.g., a VR headset 105), the rotation of the eye causes acorresponding displacement of the center of the corneal sphere 310(i.e., point 311). The tracking system 300 tracks the motion of thecenter of the corneal sphere 310 during the rotation of the eye. In someembodiments, the eye tracking system 300 generates a model for a singleeye of the user as described below in conjunction with FIG. 4. Thegenerated eye model includes eye information including but not limitedto radius and origin information for the corneal sphere 310, which isreferred herein as “eye model information.” Alternatively oradditionally, the eye model information may include radius and origininformation of the sphere that approximates a portion of the scleral ofthe eye (i.e., sphere 305). In one embodiment, the corneal motion may bemodeled as a rotation about the fixed center 306 of sphere 305. In otherembodiments, the center 306 of sphere 305 is a function of the corneaposition (modeling aspheric eye shapes, oculomotor muscle control,deformation under rotation, etc.). The generated eye model informationis stored in a database located within the system environment 100 oroutside of the system environment 100.

The control module 370 uses the stored eye models to perform one or moreoptical actions (e.g., estimate gaze direction for one or both eyes ofthe user) in the normal mode of operation while the user is viewingcontent. The control module 370 receives eye tracking information fromthe eye tracking unit 360 along with the eye model information toperform optical actions such as determining a user's gaze direction, auser's vergence angle (or vergence depth), a user's accommodation depth,identification of the user, an eye's torsional state, or somecombination thereof. The control module 370 can be implemented in eitherhardware, software, or some combination thereof.

FIG. 4 is a flowchart of an example calibration process 400 forgenerating an eye model using an eye tracking system (e.g., eye trackingsystem 300 of FIG. 3), in accordance with an embodiment. The examplecalibration process 400 of FIG. 4 may be performed by the eye trackingsystem 300, e.g., as part of a VR headset 105 and/or the VR console 110,or some other system (e.g., an AR system). Other entities may performsome or all of the steps of the process in other embodiments. Likewise,embodiments may include different and/or additional steps, or performthe steps in different orders. The example process of FIG. 4 is forgenerating an eye model for one of the user's eyes and can also beimplemented (either concurrently or sequentially) for determining an eyemodel for the user's other eye. The example process is described usingtwo illumination sources and one optical sensor for modeling the eye. Aneye model can be generated using more than two illumination sourcesand/or more than one optical sensor.

The eye tracking system 300 illuminates 410 the user's eye by turning ontwo illumination sources (e.g., illumination sources 362 and 364) thatare positioned at known locations relative to, e.g., an optical sensor(e.g., optical sensor 366). These illumination sources emit light thatis incident on the user's eye such that the cornea of the eye reflectsthe light.

The eye tracking system 300 captures 420 corneal reflections of thelight incident upon the user's eye as one or more images. In oneembodiment, the eye tracking system 300 captures a single image of thecorneal reflections. Alternatively, the eye tracking system 300 capturesmultiple images of corneal reflections as the user looks at a sequenceof known targets (e.g., specific points on the electronic display 235).

The eye tracking system 300 generates a model of the eye, where the eyemodel includes radius and origin information for the corneal sphere 310.The captured one or more images including corneal reflections of lightoriginating from the two illuminations sources at known locationsresults in a single corneal radius and a corneal sphere origin that fitsthe data of the captured images. In one embodiment, the eye trackingsystem 300 generates the eye model by using a learning model that uses areference eye. The reference eye, as referred to herein, is an eye witha reference eye model that includes known radius and origin informationfor each of corneal sphere (e.g., sphere 310) and the sphererepresenting the a portion of the scleral surface of the eye (e.g.,sphere 305). The learning model includes capturing the cornealreflections of the reference eye while the reference eye is rotated toemulate a range of human eye movements. The learning model includes therelationship between various movements of the reference eye and thecorneal reflections of the reference eye, and such relationship is usedto generate the eye model for the user's eye. For example, the cornealreflections of the user's eye captured in step 420 are compared withthat of the corneal reflections of the reference eye to extrapolatedifferent parameters for the eye model (e.g., radius and origin) for thecorneal sphere 310 of the user's eye. In such example, the radius andorigin information of the reference eye model of the learning model isextrapolated to estimate radius and origin information for the user'seye model. An example method of estimating the radius and origin for theeye model is described below.

The eye tracking system 300 determines 430 a radius of the cornealsphere 310 based on the captured images that include corneal reflectionsof at least two of the illuminating sources. Each image providessufficient information to derive an estimate of the corneal radius andcorneal position. In some embodiments, the eye model may include only asphere of fixed radius for the cornea. The corneal radius estimate maythen be refined by combining estimates over a sequence of frames usingan appropriate statistical model. For example, the samples of cornealradius may form a normal distribution about the mean over a sequence ofimages. When a sequence of images provides a set of estimations with alow variance around the mean with few outliers, the corneal radius couldbe assumed to be the mean of these samples. In embodiments with three ormore illumination sources, the difference in corneal radius predicted byeach set of two illumination sources may be used to inform anon-spherical model for the corneal surface (ex. b-splines, polynomialsurface, etc.). In yet other embodiments, the difference in cornealradius predicted in a set of frames may be combined with a predictedgaze direction per frame to inform a more complex corneal model. Inthese systems, the cornea is defined by the reference origin 311 and anorientation.

The eye tracking system 300 determines an effective center 306 of eyerotation and a distance separating the eye rotation center and thecornea center (e.g., a distance between centers 306 and 311). In oneimplementation, the eye center 306 may be assumed to be a fixed point.In this system, the user may be presented with visual targets on theelectronic display element 235 presented at known angular offsets from areference position. For example, the target may be placed at a distanceapproaching infinity with the desired angular deviation in adistortion-corrected virtual rendered space of the HMD. Thedistortion-corrected virtual rendered space is a computer-generatedthree-dimensional representation of content on the electronic displayelement 235 to a user of the HMD, where the displayed content iscorrected for distortion-based optical errors. The eye tracking system300 determines and records 440 a location of the corneal sphere whilethe user is looking toward each of these calibration targets. Thissystem may model the user's gaze as lying along the vector from the eyecenter 306 through the corneal center 311. The position of the eyecenter 306 and orientation of the eye to the virtual rendered space ofthe HMD may be inferred as the point 306 and orientation that bestsatisfies the assumption that the distance 306 to 311 remains unchangedand that the gaze vectors from 306 to recorded corneal centers 311 alignmost closely with the presented target angular offsets. In othersystems, the precise eye center 306 may be modeled as a function ofcornea position (and/or cornea orientation in more complex models) forthe purpose of gaze estimation to model physiological deviations from aspherical eye model.

Determine User Gaze Using Eye Model

FIG. 5 is a flowchart of an example process for determining a user'sgaze direction using a known eye model, in accordance with anembodiment. The example process 500 of FIG. 5 may be performed by theeye tracking system 300, e.g., as part of the VR headset 105 and/or theVR console 110 or some other system (e.g., an AR system). Other entitiesmay perform some or all of the steps of the process in otherembodiments. Likewise, embodiments may include different and/oradditional steps, or perform the steps in different orders. The exampleprocess 500 describes a normal mode of operation of the eye trackingsystem 300 where the system is tracking eye motion of a user based inpart on one or more eye models.

The eye tracking system 300 obtains 510 eye model information includingradius and origin information of corneal sphere 310. In one embodiment,the eye tracking system 300 obtains the eye model information fromwithin the eye tracking system 300 (e.g., eye tracking system 300generates the eye model as described above in conjunction with FIG. 4).Alternatively, the eye tracking system 300 obtains the eye modelinformation external to the eye tracking system 300 (e.g., a VR console110 or outside of system environment 100).

The eye tracking system 300 illuminates 520 the user's eye by turning ontwo illumination sources (e.g., illumination sources 362 and 364) thatare positioned at known locations relative to, e.g., an optical sensor(e.g., optical sensor 366). These illumination sources emit light thatis incident on the user's eye such that the cornea of the eye reflectsthe light.

The eye tracking system 300 captures 530 one or more images of theuser's cornea (i.e., corneal reflections) while the viewer is viewingcontent on a HMD (e.g., of a VR headset 105, an AR headset, or someother system using eye tracking). In one embodiment, the eye trackingsystem 300 captures a single image of the corneal reflections, where thecorneal reflections include reflections of light from the two or moreillumination sources. Alternatively, the eye tracking system 300captures multiple images of corneal reflections, where the cornealreflections include reflections of light from the two or moreillumination sources.

The eye tracking system 300 determines 540 a user's gaze direction usingthe corneal reflection data of the captured images. In one embodiment,the eye tracking system 300 extrapolates the corneal reflection datausing the obtained eye model information that includes the cornealradius and origin information. For example, the eye tracking system 300determines location data of the specific point the user is gazing uponon the electronic display 235 by casting a ray from the derived eyecenter 306 through the center 311 of the corneal sphere 310. The originand direction of this ray are transformed into a three-dimensionalcoordinate space of the distortion-corrected virtual rendered space ofthe HMD using the derived orientation of the eye.

In some embodiments, the eye tracking system 300 performs other opticalactions in addition or alternative to determining a user's gazedirection. Other optical actions include, e.g., determining a user'svergence angle (or vergence depth), a user's accommodation depth,identification of the user, an eye's torsional state, or somecombination thereof. Some of the example optical actions might requirecaptured corneal reflection data of both the user's eyes to perform theoptical actions, and one or more steps of the example process 500 mayadditionally be performed for the user's other eye (either concurrentlyor sequentially) to perform such optical actions.

Additional Configuration Information

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe patent rights to the precise forms disclosed. Persons skilled in therelevant art can appreciate that many modifications and variations arepossible in light of the above disclosure.

The language used in the specification has been principally selected forreadability and instructional purposes, and it may not have beenselected to delineate or circumscribe the inventive subject matter. Itis therefore intended that the scope of the patent rights be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thepatent rights, which is set forth in the following claims.

What is claimed is:
 1. A head mounted display (HMD) comprising: an eyetracking system including: two or more illumination sources configuredto illuminate a user's eye, a detector configured to capture lightreflected from a cornea associated with the user's eye as an image, anda control module configured to: illuminate a user's eye with light fromthe two or more illumination sources, capture, at the detector, one ormore images of the user's eye, generate a model of the eye based on thecaptured one or more images of the user's eye, and perform an opticalaction based in part on the generated model of the eye.
 2. The HMD ofclaim 1, wherein the two or more illumination sources are positionedrelative to one another such that the detector is able to capture one ormore images of the two or more illumination sources based on the lightreflected from cornea.
 3. The HMD of claim 1, wherein the two or moreillumination sources are part of the electronic display.
 4. The HMD ofclaim 1, wherein each of the two or more illumination sources areconfigured to emit light of a different wavelength.
 5. The HMD of claim1, wherein light emitted by each of the two or more illumination sourcesis modulated at a different frequency and a different amplitude.
 6. TheHMD of claim 1, wherein the two or more illumination sources form a ringpattern.
 7. The HMD of claim 1, where in the detector is an opticalsensor that can capture a plurality of images of the user's eye.
 8. TheHMD of claim 1, wherein the generated model comprises a first spherethat approximates a portion of a scleral surface of the eye and a secondsphere that approximates the cornea.
 9. The HMD of claim 8, wherein thecontrol module is further configured to: receive the one or morecaptured images of the user's eye, the images including one or morecorneal reflections; generate the model of the eye, the generated modelof the eye including a radius and an origin associated with the firstsphere and a radius and an origin associated with the second sphere; andstore the generated model in a database associated with the HMD.
 10. TheHMD of claim 9, wherein the control module is further configured tomodel corneal motion as a rotation about the origin associated with thefirst sphere.
 11. The HMD of claim 8, wherein the control module isfurther configured to: retrieve a learning model of a reference eye, thelearning model including a relationship between one or more movements ofthe reference eye and the corneal reflections associated with thereference eye; capture an image of the user's eye, the one or moreimages including a corneal reflections; and extrapolate one or moreparameter values associated with the eye model based on a comparisonbetween the one or more corneal reflections of the reference eye and thecorneal reflections associated with the captured image.
 12. The HMD ofclaim 1, wherein an optical action is selected from a group consistingof: determining a user's vergence angle, determining a user'saccommodation depth, identifying the user, determining an eye'storsional state, or some combination thereof.
 13. A head mounted display(HMD) comprising: an electronic display configured to display images tothe user; an optics block configured to magnify light received by theoptics block from the electronic display; and an eye tracking systemincluding: two or more illumination sources configured to illuminate asurface of an eye of the user, a detector configured to capture lightreflected from the cornea as an image, and a control module configuredto: illuminate a user's eye with light from the two or more illuminationsources, capture, at the detector, one or more images of the user's eye,generate a model of the eye based on the captured one or more images ofthe user's eye, and perform an optical action based in part on thegenerated model of the eye.
 14. The HMD of claim 13, wherein the two ormore illumination sources are positioned relative to one another suchthat the detector is able to capture one or more images of the two ormore illumination sources based on the light reflected from cornea. 15.The HMD of claim 13, wherein the two or more illumination sources arepart of the electronic display.
 16. The HMD of claim 13, wherein thegenerated model comprises a first sphere that approximates a portion ofa scleral surface of the eye and a second sphere that approximates thecornea.
 17. The HMD of claim 16, wherein the control module is furtherconfigured to: receive the one or more captured images of the user'seye, the images including one or more corneal reflections; generate themodel of the eye, the generated model of the eye including a radius andan origin associated with the first sphere and a radius and an originassociated with the second sphere; and store the generated model in adatabase associated with the HMD.
 18. The HMD of claim 17, wherein thecontrol module is further configured to model corneal motion as arotation about the origin associated with the first sphere.
 19. The HMDof claim 16, wherein the control module is further configured to:retrieve a learning model of a reference eye, the learning modelincluding a relationship between one or more movements of the referenceeye and the corneal reflections associated with the reference eye;capture an image of the user's eye, the one or more images including acorneal reflections; and extrapolate one or more parameter valuesassociated with the eye model based on a comparison between the one ormore corneal reflections of the reference eye and the cornealreflections associated with the captured image.
 20. The HMD of claim 13,wherein an optical action is selected from a group consisting of:determining a user's vergence angle, determining a user's accommodationdepth, identifying the user, determining an eye's torsional state, orsome combination thereof.